Protecting personal computers against a never-ending onslaught of “pestware” such as viruses, Trojan horses, spyware, adware, and downloaders on personal computers has become vitally important to computer users. Some pestware is merely annoying to the user or degrades system performance. Other pestware is highly malicious. Many computer users depend on anti-pestware software that attempts to detect and remove pestware automatically.
Anti-pestware software typically scans running processes in memory and files contained on storage devices such as disk drives, comparing them, at expected locations, against a set of “signatures” that identify specific, known types of pestware. Some kinds of pestware avoid detection, however, by using polymorphic obfuscation techniques, metamorphic obfuscation techniques, or both. Polymorphic obfuscation may involve, for example, changing the order of segments (e.g., modules or subroutines) of the pestware's program code during execution. Metamorphic obfuscation may involve, for example, changing specific program instructions to different but equivalent instructions during execution.
One possible solution to the problem of polymorphic and metamorphic obfuscation is to scan memory exhaustively for pestware signatures rather than relying on signatures being at expected locations within executable objects. Scanning memory exhaustively for every known type of pestware, however, can render a computer virtually inoperable for long periods due to the heavy processing burden. Due to the impracticality of scanning memory exhaustively, current anti-pestware software is not always able to detect pestware that employs polymorphic or metamorphic obfuscation techniques.
It is thus apparent that there is a need in the art for an improved method and system for detecting obfuscatory pestware in a computer memory.